Multimodal single-neuron, intracranial EEG, and fMRI brain responses during movie watching in human patients.

We present a multimodal dataset of intracranial recordings, fMRI, and eye tracking in 20 participants during movie watching. Recordings consist of single neurons, local field potential, and intracranial EEG activity acquired from depth electrodes targeting the amygdala, hippocampus, and medial frontal cortex implanted for monitoring of epileptic seizures. Participants watched an 8-min long excerpt from the video "Bang! You're Dead" and performed a recognition memory test for movie content. 3 T fMRI activity was recorded prior to surgery in 11 of these participants while performing the same task. This NWB- and BIDS-formatted dataset includes spike times, field potential activity, behavior, eye tracking, electrode locations, demographics, and functional and structural MRI scans. For technical validation, we provide signal quality metrics, assess eye tracking quality, behavior, the tuning of cells and high-frequency broadband power field potentials to familiarity and event boundaries, and show brain-wide inter-subject correlations for fMRI. This dataset will facilitate the investigation of brain activity during movie watching, recognition memory, and the neural basis of the fMRI-BOLD signal.

Atypical gaze patterns in autistic adults are heterogeneous across but reliable within individuals.

BACKGROUND: Across behavioral studies, autistic individuals show greater variability than typically developing individuals. However, it remains unknown to what extent this variability arises from heterogeneity across individuals, or from unreliability within individuals. Here, we focus on eye tracking, which provides rich dependent measures that have been used extensively in studies of autism. Autistic individuals have an atypical gaze onto both static visual images and dynamic videos that could be leveraged for diagnostic purposes if the above open question could be addressed. METHODS: We tested three competing hypotheses: (1) that gaze patterns of autistic individuals are less reliable or noisier than those of controls, (2) that atypical gaze patterns are individually reliable but heterogeneous across autistic individuals, or (3) that atypical gaze patterns are individually reliable and also homogeneous among autistic individuals. We collected desktop-based eye tracking data from two different full-length television sitcom episodes, at two independent sites (Caltech and Indiana University), in a total of over 150 adult participants (N = 48 autistic individuals with IQ in the normal range, 105 controls) and quantified gaze onto features of the videos using automated computer vision-based feature extraction. RESULTS: We found support for the second of these hypotheses. Autistic people and controls showed equivalently reliable gaze onto specific features of videos, such as faces, so much so that individuals could be identified significantly above chance using a fingerprinting approach from video epochs as short as 2 min. However, classification of participants into diagnostic groups based on their eye tracking data failed to produce clear group classifications, due to heterogeneity in the autistic group. LIMITATIONS: Three limitations are the relatively small sample size, assessment across only two videos (from the same television series), and the absence of other dependent measures (e.g., neuroimaging or genetics) that might have revealed individual-level variability that was not evident with eye tracking. Future studies should expand to larger samples across longer longitudinal epochs, an aim that is now becoming feasible with Internet- and phone-based eye tracking. CONCLUSIONS: These findings pave the way for the investigation of autism subtypes, and for elucidating the specific visual features that best discriminate gaze patterns-directions that will also combine with and inform neuroimaging and genetic studies of this complex disorder.

A Cautionary Note on Predicting Social Judgments from Faces with Deep Neural Networks.

People spontaneously infer other people's psychology from faces, encompassing inferences of their affective states, cognitive states, and stable traits such as personality. These judgments are known to be often invalid, but nonetheless bias many social decisions. Their importance and ubiquity have made them popular targets for automated prediction using deep convolutional neural networks (DCNNs). Here, we investigated the applicability of this approach: how well does it generalize, and what biases does it introduce? We compared three distinct sets of features (from a face identification DCNN, an object recognition DCNN, and using facial geometry), and tested their prediction across multiple out-of-sample datasets. Across judgments and datasets, features from both pre-trained DCNNs provided better predictions than did facial geometry. However, predictions using object recognition DCNN features were not robust to superficial cues (e.g., color and hair style). Importantly, predictions using face identification DCNN features were not specific: models trained to predict one social judgment (e.g., trustworthiness) also significantly predicted other social judgments (e.g., femininity and criminal), and at an even higher accuracy in some cases than predicting the judgment of interest (e.g., trustworthiness). Models trained to predict affective states (e.g., happy) also significantly predicted judgments of stable traits (e.g., sociable), and vice versa. Our analysis pipeline not only provides a flexible and efficient framework for predicting affective and social judgments from faces but also highlights the dangers of such automated predictions: correlated but unintended judgments can drive the predictions of the intended judgments. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1007/s42761-021-00075-5.

Four dimensions characterize attributions from faces using a representative set of English trait words.

People readily (but often inaccurately) attribute traits to others based on faces. While the details of attributions depend on the language available to describe social traits, psychological theories argue that two or three dimensions (such as valence and dominance) summarize social trait attributions from faces. However, prior work has used only a small number of trait words (12 to 18), limiting conclusions to date. In two large-scale, preregistered studies we ask participants to rate 100 faces (obtained from existing face stimuli sets), using a list of 100 English trait words that we derived using deep neural network analysis of words that have been used by other participants in prior studies to describe faces. In study 1 we find that these attributions are best described by four psychological dimensions, which we interpret as "warmth", "competence", "femininity", and "youth". In study 2 we partially reproduce these four dimensions using the same stimuli among additional participant raters from multiple regions around the world, in both aggregated and individual-level data. These results provide a comprehensive characterization of trait attributions from faces, although we note our conclusions are limited by the scope of our study (in particular we note only white faces and English trait words were included).

Cortical networks of dynamic scene category representation in the human brain.

Humans have an impressive ability to rapidly process global information in natural scenes to infer their category. Yet, it remains unclear whether and how scene categories observed dynamically in the natural world are represented in cerebral cortex beyond few canonical scene-selective areas. To address this question, here we examined the representation of dynamic visual scenes by recording whole-brain blood oxygenation level-dependent (BOLD) responses while subjects viewed natural movies. We fit voxelwise encoding models to estimate tuning for scene categories that reflect statistical ensembles of objects and actions in the natural world. We find that this scene-category model explains a significant portion of the response variance broadly across cerebral cortex. Cluster analysis of scene-category tuning profiles across cortex reveals nine spatially-segregated networks of brain regions consistently across subjects. These networks show heterogeneous tuning for a diverse set of dynamic scene categories related to navigation, human activity, social interaction, civilization, natural environment, non-human animals, motion-energy, and texture, suggesting that the organization of scene category representation is quite complex.

Motor and sensory features successfully decode autism spectrum disorder and combine with the original RDoC framework to boost diagnostic classification.

Sensory processing and motor coordination atypicalities are not commonly identified as primary characteristics of autism spectrum disorder (ASD), nor are they well captured in the NIMH's original Research Domain Criteria (RDoC) framework. Here, motor and sensory features performed similarly to RDoC features in support vector classification of 30 ASD youth against 33 typically developing controls. Combining sensory with RDoC features boosted classification performance, achieving a Matthews Correlation Coefficient (MCC) of 0.949 and balanced accuracy (BAcc) of 0.971 (p = 0.00020, calculated against a permuted null distribution). Sensory features alone successfully classified ASD (MCC = 0.565, BAcc = 0.773, p = 0.0222) against a clinically relevant control group of 26 youth with Developmental Coordination Disorder (DCD) and were in fact required to decode against DCD above chance. These findings highlight the importance of sensory and motor features to the ASD phenotype and their relevance to the RDoC framework.

No strong evidence that social network index is associated with gray matter volume from a data-driven investigation.

Recent studies in adult humans have reported correlations between individual differences in people's Social Network Index (SNI) and gray matter volume (GMV) across multiple regions of the brain. However, the cortical and subcortical loci identified are inconsistent across studies. These discrepancies might arise because different regions of interest were hypothesized and tested in different studies without controlling for multiple comparisons, and/or from insufficiently large sample sizes to fully protect against statistically unreliable findings. Here we took a data-driven approach in a pre-registered study to comprehensively investigate the relationship between SNI and GMV in every cortical and subcortical region, using three predictive modeling frameworks. We also included psychological predictors such as cognitive and emotional intelligence, personality, and mood. In a sample of healthy adults (n = 92), neither multivariate frameworks (e.g., ridge regression with cross-validation) nor univariate frameworks (e.g., univariate linear regression with cross-validation) showed a significant association between SNI and any GMV or psychological feature after multiple comparison corrections (all R-squared values ≤ .1). These results emphasize the importance of large sample sizes and hypothesis-driven studies to derive statistically reliable conclusions, and suggest that future meta-analyses will be needed to more accurately estimate the true effect sizes in this field.

Spatially informed voxelwise modeling for naturalistic fMRI experiments.

Voxelwise modeling (VM) is a powerful framework to predict single voxel responses evoked by a rich set of stimulus features present in complex natural stimuli. However, because VM disregards correlations across neighboring voxels, its sensitivity in detecting functional selectivity can be diminished in the presence of high levels of measurement noise. Here, we introduce spatially-informed voxelwise modeling (SPIN-VM) to take advantage of response correlations in spatial neighborhoods of voxels. To optimally utilize shared information, SPIN-VM performs regularization across spatial neighborhoods in addition to model features, while still generating single-voxel response predictions. We demonstrated the performance of SPIN-VM on a rich dataset from a natural vision experiment. Compared to VM, SPIN-VM yields higher prediction accuracies and better capture locally congruent information representations across cortex. These results suggest that SPIN-VM offers improved performance in predicting single-voxel responses and recovering coherent information representations.